-
Notifications
You must be signed in to change notification settings - Fork 385
RBAC setup behind the aggregator. #936
RBAC setup behind the aggregator. #936
Conversation
resolves #880 |
@kibbles-n-bytes is rbac on in jenkins CI? I'm not sure why this would fail over there. |
contrib/svc-cat-rbac.yaml
Outdated
- apiGroup: "" | ||
kind: ServiceAccount | ||
name: server | ||
namespace: catalog |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this namespace right or does it need to be parameterized to match SVCCAT_NAMESPACE
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Unspecified assumption currently. Needs to be integrated into the helm chart to get parameterized.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@MHBauer It would be best parameterized in the values.yaml file. Ping me if you want help with that
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@MHBauer I think I answered a few helm questions, and I do think the namespace should at least be parameterized in the helm chart (even though that's separate from the shell scripts, it's a step in the right direction).
Also, I had a few additions to the markdown file.
Overall, this is great.
contrib/svc-cat-rbac.yaml
Outdated
- apiGroup: "" | ||
kind: ServiceAccount | ||
name: server | ||
namespace: catalog |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
After https://github.com/kubernetes-incubator/service-catalog/pull/936/files#r121578665 is done, this should read:
namespace: {{ .Values.namespace }}
contrib/svc-cat-rbac.yaml
Outdated
- apiGroup: "" | ||
kind: ServiceAccount | ||
name: server | ||
namespace: catalog |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same here RE the parameterized namespace. There are some other similar changes below, but I won't comment on them all.
docs/rbac-setup.md
Outdated
# RBAC Setup | ||
|
||
If rbac is enabled, allow the service catalog to grant access for it's | ||
service token. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@MHBauer could you add a bit here on what would have been set in the original helm install
command, so that people know whether they enabled RBAC? It may not be obvious to a newcomer.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@arschles Gone!
docs/rbac-setup.md
Outdated
|
||
## Install our rbac rules | ||
|
||
See the [script in contrib](../contrib/rbac-setup.sh). See the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should they run the script in contrib to install the RBAC rules? If so, can you add that little piece to the docs
@@ -20,6 +20,7 @@ spec: | |||
release: "{{ .Release.Name }}" | |||
heritage: "{{ .Release.Service }}" | |||
spec: | |||
serviceAccountName: server |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this and the serviceAccountName
in controller-manager-deployment.yaml
would be worth parameterizing in the values.yaml
file as well
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The walkthrough test fails because contrib/rbac-setup.sh
isn't run in contrib/hack/test_walkthrough.sh
, so the service account additions here break the deployment.
We should either make these serviceAccountName
s optional, or update both contrib/hack/test_walkthrough.sh
and the walkthrough docs to require running contrib/rbac-setup.sh
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sheesh. You mentioned it and tagged it for me and I still missed it. sorry guys.
@MHBauer I was out earlier this week, looking into this today. |
@@ -20,6 +20,7 @@ spec: | |||
release: "{{ .Release.Name }}" | |||
heritage: "{{ .Release.Service }}" | |||
spec: | |||
serviceAccountName: server |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The walkthrough test fails because contrib/rbac-setup.sh
isn't run in contrib/hack/test_walkthrough.sh
, so the service account additions here break the deployment.
We should either make these serviceAccountName
s optional, or update both contrib/hack/test_walkthrough.sh
and the walkthrough docs to require running contrib/rbac-setup.sh
.
contrib/rbac-setup.sh
Outdated
# See the License for the specific language governing permissions and | ||
# limitations under the License. | ||
|
||
SVCCAT_NAMESPACE=catalog |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This assumes the namespace will be "catalog". This should probably be parameterized. A simple way would be:
SVCCAT_NAMESPACE="${SVCCAT_NAMESPACE:-catalog}"
contrib/rbac-setup.sh
Outdated
kubectl create namespace ${SVCCAT_NAMESPACE} | ||
kubectl create serviceaccount server --namespace=${SVCCAT_NAMESPACE} | ||
kubectl create serviceaccount controller --namespace=${SVCCAT_NAMESPACE} | ||
kubectl create -f svc-cat-rbac.yaml |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This assumes we're running this file while in contrib/
. I'd add
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
to the top of the file, and then change the file reference to "${ROOT}/contrib/svc-cat-rbac.yaml
I agree that the account names need to be parameterized here. The account names need to match the rbac definition yaml. The service account creation should become part of the charts. The rbac yaml should become part of the charts. |
contrib/jenkins/init_cluster.sh
Outdated
--clusterrole=cluster-admin --user="${ACCOUNT_NAME}" \ | ||
|| { echo 'Cannot not create cluster-admin role for service account.'; exit 1; } | ||
|
||
${ROOT}/contrib/rbac-setup.sh \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
maybe won't need this script at all.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since you got rid of it now, lines 66-68 should be removed as well.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am really dumb some times.
I think we're good now. rbac by default. At the very least it shouldn't break anything on someone running without rbac on. |
@@ -20,6 +20,7 @@ spec: | |||
release: "{{ .Release.Name }}" | |||
heritage: "{{ .Release.Service }}" | |||
spec: | |||
serviceAccountName: controller |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You still need to templatize the definition here and in apiserver-deployment.yaml
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🦈 good eye. I'll take another pass through. Thanks!
@kibbles-n-bytes @arschles I can't believe you both tagged some things and I still overlooked them. Sorry about that. We should be good now. Not sure why jenkins is failing. Looks to me like it successfully installs. |
@MHBauer is the jenkins failure a flake? |
if you're testing with an apiserver build from master, kubernetes/kubernetes#46812 (comment) would have broken this a couple days ago |
@liggitt our apiserver stuff is somewhere between levels v1.5.5 and v1.6.0-rc1 |
undo! |
looking at https://service-catalog-jenkins.appspot.com/job/service-catalog-PR-testing2/885/
which would be a flake to me. |
@MHBauer As I mentioned on Slack, I believe that's the same issue I hit when trying to run this on GKE. I had to create a ClusterRoleBinding with the "cluster-admin" role before doing any of the RBAC stuff. |
# cluster roles. Needed for RBAC setup. | ||
ACCOUNT_NAME="$(gcloud info | grep Account | sed 's/.*\[\(.*\)\]/\1/')" | ||
kubectl create clusterrolebinding jenkins-cluster-admin-binding \ | ||
--clusterrole=cluster-admin --user="${ACCOUNT_NAME}" \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@arschles @kibbles-n-bytes for default minikube I did
kubectl create clusterrolebinding minikube-cluster-admin-binding --clusterrole=cluster-admin --user=minikube
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nope, did not help, same
Error: release catalog failed: clusterroles.rbac.authorization.k8s.io "servicecatalog.k8s.io:apiserver" is forbidden: attempt to grant extra privileges: [PolicyRule{Resources:["namespaces"], APIGroups:[""], Verbs:["get"]} PolicyRule{Resources:["namespaces"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["namespaces"], APIGroups:[""], Verbs:["watch"]}] user=&{system:serviceaccount:kube-system:default d6100657-632b-11e7-8c06-befc308e6091 [system:serviceaccounts system:serviceaccounts:kube-system system:authenticated] map[]} ownerrules=[] ruleResolutionErrors=[]
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
rbac wasn't on in minikube, oops.
add --extra-config=apiserver.Authorization.Mode=RBAC
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can add this to the walkthrough docs in a follow-up
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Tracked in #1021
@MHBauer Still investigating this failure, should have a fix out soon. |
@kibbles-n-bytes rebased on to 34ed5cd. |
@MHBauer IMO it's fine to leave these |
@MHBauer ready to review and merge? |
@@ -0,0 +1,127 @@ | |||
apiVersion: v1 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@MHBauer I believe we should be able to use the following to test whether the rbac API group is available:
{{- if .Capabilities.APIVersions.Has "rbac.k8s.io/v1beta1" }}
// stuff
{{- end }}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looking great, just a couple suggestions / gaps to close.
@@ -0,0 +1,127 @@ | |||
apiVersion: v1 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@MHBauer I believe we should be able to use the following to test whether the rbac API group is available:
{{- if .Capabilities.APIVersions.Has "rbac.k8s.io/v1beta1" }}
// stuff
{{- end }}
charts/catalog/templates/rbac.yaml
Outdated
items: | ||
|
||
# TODO: if this is just for namespace lifecycle admission, move to a generic role | ||
- apiVersion: rbac.authorization.k8s.io/v1beta1 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe this should be the only use for the API server, and does constitute what is necessary for namespace lifecycle admission. I'm cool calling this role something like namespace-lifecycle-admission
, your choice.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there more to do to address the todo? or a followup.
charts/catalog/templates/rbac.yaml
Outdated
# TODO: do not grant global access, limit to particular secrets referenced from bindings | ||
- apiGroups: [""] | ||
resources: ["secrets"] | ||
verbs: ["get","create"] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The controller manager deletes secrets when bindings are deleted, so delete is definitely needed.
charts/catalog/templates/rbac.yaml
Outdated
resources: ["secrets"] | ||
verbs: ["get","create"] | ||
- apiGroups: [""] | ||
resources: ["configmaps"] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think we have any uses of configmaps in the code today, do we? Does auth delegation rely on it?
charts/catalog/templates/rbac.yaml
Outdated
resources: ["secrets"] | ||
verbs: ["get","create"] | ||
- apiGroups: [""] | ||
resources: ["configmaps"] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(It might be handy to comment saying what each of these permissions is used for)
charts/catalog/templates/rbac.yaml
Outdated
- apiGroups: [""] | ||
resources: ["namespaces"] | ||
verbs: ["get", "list", "watch"] | ||
- apiVersion: rbac.authorization.k8s.io/v1beta1 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks like we are missing RBAC rules for settings API group
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What should that look like?
I know that the 'requestheader-ca' information is looked up by the apiserver in a configmap. I know for api aggregation it's needed. Maybe for auth delegation as well. Any comment @DirectXMan12 ? |
What's the proper way to drop comments in? Is there a way to get the comment into kube so we know what it does later? I obviously want a very specific name, but there's only so many characters I can use in there. |
The
I think commenting in the yaml spec is good enough. |
I see that now. This was always intended as a rough draft of "what happened" when I ran some tests with the rbac in permissive mode. I don't know of any more definitive way to determine the exact permissions we may need. This is a bad UX. I'll take configmaps out and test, but it's going to delay this again. |
@kibbles-n-bytes rebase why? |
gotcha gotch. I'll do some refactoring of the commits while I'm rebasing. |
/retest |
@MHBauer Check my latest comment on |
@kibbles-n-bytes thanks |
yaml was generated by putting rbac into permissive mode and seeing what it would have denied. then @liggitt processed the log messages to create the output and left some suggested TODOs.
@pmorie do you undestand the leader election TODOs?
Service accounts were done to see what each server was specificly doing. We can back that out and adapt the yaml.
Need to have the auth setup as in the auth docs.